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ABSTRACT 

The purpose of this study is to develop scientific literacy evaluation instrument that tested 
its validity, reliability, and characteristics to measure the skill of student’s scientific 
literacy used four scientific literacy, categories as follow: science as a body of knowledge 
(category A), science as a way of thinking (category B), science as a way of investigating 
(category C), and the interaction between science, technology, and society (category D). 
The subjects of this developing study were 9 th grade student of junior high schoolin Kudus. 
Validity test is done by the content, construct, and concurrent validation. Reliability 
testing is done by calculating a reliability coefficient “r”. Characteristics tested by 
determining the level of difficulty, distinguishing features, and the proportion or ratio of 
scientific literacy category in the instrument. Profile of scientific literacy ability is 
determined by measuring students' mastery of scientific literacy. The result of reliability 
test is the price of“r” at first trial is 0.59 and the last trial is 0.74. Validity test results 
indicate that the instrument meets the content, construct, and concurrent validity with 
valid, very valid, and valid category. Characteristics of evaluation instruments developed 
shows that the instrument has a level of difficulty that the proportion of about 13% easy, 
67% about the medium, and 20% about the difficult. The results of the analysis of 
discrimination power combined with a level of difficulty and obtained 5 items with poor 
quality and should be discarded. Comparison category A: B: C: D is 7: 2: 3: 3. Profile of 
literacy skills mastery of science shows that scientific literacy is still low at below 50% for 
all categories. Evaluation instruments developed meet the content validity, construct, and 
alignment with valid, very valid, and valid criteria. 
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Introduction 

In the 21st century, the progress of science and technology in various 
countries is rapidly increasing. The key for the advancement of science education 
including the quality of applied learning in each country. Science education as one 
of the subjects in junior high school became an important foundation in the form 
of qualified human resources. Based on the opinions expressed in Rusilowati 
(2013), building science education students to think in understanding natural 
phenomena or events with scientific methods as do scientists. However, science 
educationis stillhaving less attention in learning in the country. 

The quality of education, especially science education in Indonesia is still low 
when compared to other developing countries. Weak education in Indonesia, 
especially science education indicated by the low level of scientific literacy 
achievement in PISA (Program for International Student Assessment). Indonesia 
has always obtained a score below the average score. In 2000, the science subjects 
put Indonesia on order 38 out of 41 participating countries. In 2003, Indonesia 
was ranked 38 out of the 40 participants. In 2006 the number of participating 
countries increased, Indonesia was ranked 50 out of 57 countries, while 60 out of 
65 countries obtained Indonesia in 2009. Based on the data from PISA 2012, 
Indonesia received ratings 64 of the 65 participating countries with an average 
acquisition value of science literacy component of Indonesian children was 382 
(OECD, 2014: 5). 

Achievement level of scientific literacy Indonesia for 12 years participation 
has always ranked fifth bottom when scientific literacy is very important in 
determining the quality of education in a country (Hayat & Yusuf, 2011). The 
scientific literacy is very important mastered by students in relation to how they 
perceive the environment, health, economy, and the problems of modern society 
more dependent on technology and progress and development of science. 
Therefore, measurement of scientific literacy is very important to know the extent 
to which the student has literated science thus improving the quality of education 
in Indonesia can be done and can compete with other countries. 

Preparation of evaluation instruments based scientific literacy is one way to 
measure the literacy skills of students, especially in science or science. 
Sulistiawati (2015) stated that in order to measure the students' science literacy 
skills can be used on a few questions from PISA. Holbrook & Rannikmae (2009) 
state that the emphasis on enhancing scientific literacy is placed on an 
appreciation of nature of science, the development of personal attributes and the 
acquisition of socioscientific skills and values. Chiappetta et al. (1991) stated that 
there are four categories of scientific literacy that science as a body of knowledge, 
science as a way of thinking, science as a way of Investigating, and the interaction 
between science, technology, and society. Rusilowati, et al (2015) expand the 
catogory of sciencetific literacy of interaction science with environment. 

Based on those statements, it is necessary to develop scientific literacy-based 
evaluation instruments to measure students' science literacy skills with the 
theme of the cycle. The materials are selected based on relation to the theme of 
the cycle and refers to the matter of PISA. Evaluation instrument based scientific 
literacy is already there but is still limited. Through the development of 
evaluation instruments is expected educators can measure students' science 
literacy skills and familiarize students with problems of international standard 
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in learning in Indonesia so that to support the improvement of the quality of 
education at the world level, particularly in the ability of science literacy. 

Purpose 

The purpose of this study is to develop scientific literacy evaluation 
instrument that tested its validity, reliability, and characteristics to measure the 
skill of student’s scientific literacy used four scientific literacy, categories as 
follow:science as a body of knowledge, science as a way of thinking science as a 
way of investigating, and the interaction between science, technology, and society. 

Method 

Design of this developing study is used Borg & Gall design (2007) that have 
been simplified into three main stages, namely: (a) the preliminary study stage; 
(B) the stage of development studies; (C) the evaluation stage. 

The preliminary study stage includes the study of literature and field of 
study. The study of literature was done by searching the literature or data relating 
to science literacy and science literacy skills while field of studies to obtain facts 
or findings in the field. Furthermore, the study phase development includes the 
first product design, test validation by experts, analysis and revision, first trial, 
analysis and improvement, until the product generates hypothetical. The last 
stage is the evaluation includes product testing hypothetical at the end of the trial 
until the final product is produced. 

The validity test includes content, construct, and concurrent validity. The 
content validity test was done through the study of the instrument by the 
supervisor 1 as the evaluation expert and supervisor 2 as science matter expert, 
construct validity test performed with validation techniques by expert (judgment 
experts) through questionnaires validation. The validity of concurrent was used 
to determine the correlation between matter developed with the question of the 
original PISA uses a technique product moment correlation (r X y). 

Reliability testing was done by calculating the coefficient of reliability. Test 
of characteristics by determining the level of difficulty, discrimination power, and 
the proportion or ratio of scientific literacy category in the instrument. 

The developed instrument used to measure students' scientific literacy skills. 
The ability of science literacy test was done by calculating the percentage of 
mastery of scientific literacy based on Mardapi (2012) which has been combined 
with four categories of scientific literacy. 

Results and Discussion 

Validity Evaluation Instrument 

The content validity test was done before the evaluation instrument based 
scientific literacy tested in the first trial to obtain suggestions for improvements. 
Based on the review experts, the developed instrument that was categorized valid 
with the revision. 

The result of construct validity test which obtained by calculating a score of 
expert validation in the questionnaire can be seen in Table 1. Based on the results 
shown in Table 1 it can be concluded that the evaluation instrument developed 
based on scientific literacy is very valid. 
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Table 1. The result of the validation of evaluation instruments based on scientific literacy by 
experts 


No 

Aspect 

Validator 1 

Validator II 

Score (%) 

Average 

Result 

1 

Material 

96 

95 

95,5 

Very Valid 

2 

Construction 

75 

75 

75 

Medium 

3 

Language 

97 

95 

96 

Very Valid 


Construct 



88,8 

Very Valid 


validity 






The suggestion that was given by some experts is application of a picture 
must be able to explain the question not only for decoration, application of 
question’s sentences must be effective and the indicator of lattice matter must be 
fixed with the indicator of science literacy. 

The measurement of alignment validity was done with the correlation 
technique Product-Moment. After doing the data analysis, the result will be 
obtained as shown on r xy calculation is 0.77 with valid category. 

The result of measurement shows that the alternative hypothesis accepted, 
there is a high correlation between PISA question with the question based on 
science literacy that obtained. If the correlation index has the value +1,00 it 
means that the correlation has a positive way. 

The instrument has a high alignment validity it is due to some factors, there 
are, (1) The ability of students adequate; (2) The objectivity of teachers in the 
giving of scores; (3) and the students are on the same level in a group while being 
tested with the PISA question and question of the development. 

According Subiyanto (1988), subjective teacher for students' attitudes may 
affect the validity or the validity of the instrument. Such attitudes can indeed be 
considered in the evaluation, but clearly should not affect the measurement 
results. 

Reliability of Evaluation Instrument 

Reliability tests performed after first trial of scientific literacy instruments 
to the 32 test subjects. Reliability test result is 0.59. Researchers testing out the 
product in the form of 20 items based on scientific literacy that have been 
validated by experts. 

At the end of the test, the test instrument based scientific literacy 
accompanied by the original PISA instrument as a comparison analysis of 
reliability (a = 5%) in order to obtain the results presented in Table 2. 


Table 2. Result of reliability test data analyzing in the final trial 


N 

Instrument 

^calculation 

Ttable 

Criteria 

60 

Instrument ‘s development 

0.72 

0.254 

Reliable 

60 

Instrument PISA 

0.44 

0.254 

Reliable 


In Table 2 the price is greater than r table. Thus, it can be concluded that the 
evaluation instrument based scientific literacy tested is reliable. 
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PISA instrument reliability level is low due to several factors, including: (1) 
PISA items appropriate to the theme are only 6 questions, and it will be explained 
on the limitations of the study; (2) The ability of the students are not the same 
with each other. The reason is based on the factors put forward by Grounlund as 
quoted by Arifin (2009) that the length of the test will affect the reliability of a 
test. 

Characteristics of Evaluation Instrument 

Characteristics of scientific literacy instruments developed include the level 
of difficulty, discrimination power, and the proportion of scientific literacy 
category show at Table 3. 

(1) Level of Difficulty (LD) and Discrimination Power (DP) 


Table 3. Level of Difficulty and Discrimination Power 


Item 

LD 

DP 

Interpretation 

1 

0.35 

0.23 

Item Received with revision 

2 

0.72 

0.40 

Item Received 

3 

0.45 

0.20 

Item Received with revision 

4 

0.51 

0.42 

Item Received 

5 

0.26 

0.05 

Item Rejected 

6 

0.26 

0.20 

Item Received with revision 

7 

0.55 

0.20 

Item Received with revision 

8 

0.31 

0.28 

Item Received with revision 

9 

0.39 

0.58 

Item Received 

10 

0.23 

0.03 

Item Rejected 

11 

0.23 

0.00 

Item Rejected 

12 

0.45 

0.20 

Item Received with revision 

13 

0.52 

0.06 

Item Rejected 

14 

0.54 

0.55 

Item Received 

15 

0.59 

0.22 

Item Received with revision 

16 

0.20 

0.25 

Item Received with revision 

17 

0.76 

0.23 

Item Received with revision 

18 

0.42 

0.06 

Item Rejected 

19 

0.66 

0.25 

Item Received with revision 

20 

0.59 

0.22 

Item Received with revision 


(2) Proportion of Scientific Literacy Category 

The proportion of each category in the scientific literacy test instrument 
developed by researchers are presented in Table 4. 

Based on the table, the category of scientific literacy in the instrument of 
development and about the PISA results native has a different comparison. 
Instrument based scientific literacy as the initial product has a ratio of category 
A: B: C: D of 8: 4: 4: 4 in proportion to Wilkinson et al. (1999) which states that 
the proportion of scientific literacy categories are balanced meet the criteria of 
scientific literacy in the ratio 2: 1: 1: 1 for each aspect of scientific literacy. Having 
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been tested on a limited scale, obtained a new comparison of 15 questions were 
selected, namely Category A: B: C: D by 7: 2: 3: 3. The comparison is not in 
accordance with the comparison Wilkinson because of several issues which have 
been analyzed have poor quality items so it must be removed and this affects 
scientific literacy category comparison before and after tested. Unlike the matter 
of development results, PISA questions were used in this study only meet the 
category A and B due to the limitations of the original PISA questions provided. 
Most of the original PISA questions were found to be in the category B. 


Table 4. Proportion of Scientific Literacy Category 


No 

Scientific 

Literacy 

Category 

Developed Instrument 

In the initial trial 

Developed 

Instrument 

In the final trial 

PISA Instrument 

Items 

Percentage 

Items 

Percentage 

Items 

Percentage 

1 


A 

8 

40% 

7 

47% 

2 

33% 

2 


B 

4 

20% 

2 

13% 

4 

67% 

3 


C 

4 

20% 

3 

20% 

- 

- 

4 


D 

4 

20% 

3 

20% 

- 

- 


Note: 

A= Science as a body knowledge 
B= Science as a way of thinking 
C= Science as a way of investigating 

D= Interaction of science, environment, technology, and society 


The Profile of Students Scientific Literacy Skills 

Students' science literacy skills can be measured by analyzing the student's 
mastery of each category of scientific literacy. Mastery of scientific literacy of 
students is measured by looking at the test results matter science literacy-based 
development results and PISA test questions in each category of scientific literacy. 
The results of the analysis about the results showed that the mastery of 
competencies development category A is equal to 29.88%, amounting to 38.13% of 
category B, category C 49.03%, and 28.61% for competence mastery Mastery 
Category D. scientific literacy students diagram presented Figure 1. 

In addition to the developed instrument results, students' science literacy 
skills can also be seen from the mastery of scientific literacy PISA original use 
problems. Mastery of scientific literacy category A category amounted to only 
32.15% and are not much different from the mastery of category B, which only 
reached 32.5%. The results of the analysis of scientific literacy abilities can be 
explained using a bar chart is presented in Figure 2. 

(1) Science as a body knowledge 

In this category, the achievement of scientific literacy on PISA questions is 
higher than the mastery of the developed instrument due to the number of 
developed items in this category have more varied material also compared to 
about PISA so that the possible answers any more. 
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Science as a body Science as a way of Science as a way of Interaction of science, 


kowledge thinking investigating technology, and society 

Scientific Literacy Categories 


Figure 1. Mastery of Science Literacy in Development Instrument Results 


33 



Science as a body knowledge Science as a way of thinking 

Scientific Literacy Categories 


Figure 2. Mastery of Science Literacy in PISA Instrument Results 


Possible scientific literacy in this category that still has not reached 50%, 
namely: (1) Teachers often teach formulas compared with the concept; (2) 
Students do not understand the basic concept being taught by the teacher; (3) 
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Students do not have knowledge of the facts, terminology and concepts of science 
enough. Results of research Odja & Payu (2014) showed that the students' science 
literacy skills that are in categories where students agree or disagree with a 
statement or issue but can not give a scientific explanation. 

(2) Science as a way of thinking 

The results showed an average yield relatively low on both questions. In 
connection with these results it can be said that the students' skills in critical 
thinking, inductive deductive reasoning, analyzing causality and analyze 
scientific data are lacking. 

(3) Science as a way of investigating 

Results obtained in line with research Odja & Payu (2014) showed that junior 
high school students who studied the average yet have the ability to communicate 
the results of the experiment. 

Factors causing low mastery of students on science as a way of investigating 
categories namely: (1) Students rarely doing lab activities; (2) Students do not 
understand these terms in some scientific investigation activities such as 
independent variables and the dependent variable; (3) Students spend more time 
with science that promotes rote learning. According to Leonard, was quoted as 
saying by Maturradiyah & Rusilowati (2015), a science lesson should be more 
emphasis on student activity, reducing memorizing knowledge, more emphasis on 
science process skills to get the concept, and most of the time students spent in 
the laboratory or field work. 

(4) Interaction of science, environment, technology, and society 

Compared to other categories, about the interaction of science, technology, 
and society category is the least mastered by students. These results are in 
accordance with the students' answers during the interview stating that they lack 
an understanding of the application of science and technology in everyday life. 
The level of scientific literacy abilities are lower in these categories indicate a lack 
of knowledge of students in science and technology.The importance of mastering 
this category is supported by the opinion of Ibrahim & Aspar (2011: 9) states that 
without the knowledge of good science, we will become weak and technology users 
are not able to apply all of the existing technological sophistication. Based on 
Figure 1 and 2, in general it can be said that the students' science literacy 
remained low in all categories of scientific literacy is below 50%. Results obtained 
in line with OECD data (OECD, 2014: 5) placing Indonesia ranked 64th out of 65 
participating countries with an average acquisition value of science literacy 
component of Indonesian children by 382. In addition, Ridwan et al. (2013) in his 
study also concluded the same thing: the ability SMP studied science literacy is 
at the functional level where these levels belong in science literacy skills are 
pretty low. 

Conclusion 

Evaluation instruments developed meet the content validity, construct, and 
alignment with valid, very valid, and valid criteria. Reliability of the instrument 
of 0.59 in initial trials and 0.72 at the end of the trial. The proportion difficulty 
level obtained by the evaluation instrument has met the 13% ratio easy items, 
about 67% moderate, and 20% about the difficult. Discrimination power combined 
with a level of difficulty is used to determine the quality of the items used. 
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Comparison category A: B: C: D, respectively, are 7: 2: 3: 3. In general, the profile 
of students' science literacy is low as indicated by the percentage of mastery of 
scientific literacy is below 50% for each category. 

Associated with a lower ability students 'science literacy, the authors suggest 
the need to develop scientific literacy-based evaluation instrument that can 
measure students' science literacy skills so that students are familiar with the 
problems based on scientific literacy. 
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